Skip to content

feat: collect the OTLP data exported by the benchmarks - #589

Draft
lvaroqui wants to merge 4 commits into
mainfrom
cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet
Draft

lvaroqui wants to merge 4 commits into
mainfrom
cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet

Conversation

@lvaroqui

Copy link
Copy Markdown
Contributor

Add an experimental --experimental-otel flag (CODSPEED_EXPERIMENTAL_OTEL) that collects the OpenTelemetry data the benchmarks export, and uploads it with the results.

When the flag is set, each run part starts an OTLP/HTTP receiver on a free localhost port and passes its URL to the benchmark in CODSPEED_OTEL_ENDPOINT. The receiver accepts protobuf and JSON exports of traces, logs and metrics, plain or gzipped, on /v1/{traces,logs,metrics}.

While the benchmark runs, the receiver only writes each request body to disk as received, so it adds no compression work next to the measured process. A request is stored before it is acknowledged, so everything the process sent before exiting is kept. Once the process exits, the requests are decompressed and packed into a single otel.tar.zst in the profile folder. Compressing them as one stream with long-distance matching takes advantage of the content that repeats across export batches. Nothing is written when the benchmark exports nothing.

To put the spans and logs on the same clock as the benchmark markers, instrument-hooks is bumped for instrument_hooks_monotonic_offset: integrations timestamp with the monotonic clock shifted to read as wall clock, and record the offset in the codspeed.clock.monotonic_offset_ns resource attribute. The bindings expose it as InstrumentHooks::monotonic_offset and MONOTONIC_OFFSET_ATTRIBUTE.

tests/otel.rs runs the divan otel_example end to end through codspeed run --experimental-otel in walltime mode, and checks that every span and log record, shifted back by its resource's offset, falls within the benchmark markers. It needs a codspeed-rust checkout next to this repository, so it is ignored by default: cargo test --test otel -- --ignored. Set CODSPEED_OTEL_TEST_PROFILE_FOLDER to keep the profile folder for inspection.

Stacked on #587.

Closes COD-3863

@lvaroqui
lvaroqui added this pull request to stack #590 October 16, 2026 12:56
@codspeed

codspeed Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 8 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

✅ 37 untouched benchmarks
⏩ 10 skipped benchmarks1


Comparing cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet (cf0d278) with main (53c7660)

Open in CodSpeed

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Base automatically changed from cod-3863-chore-avoid-blocking-in-runner to main October 16, 2026 13:07
@lvaroqui
lvaroqui force-pushed the cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet branch 2 times, most recently from 1ff3345 to 33db967 Compare October 16, 2026 16:14
lvaroqui and others added 4 commits October 20, 2026 16:07
Bump instrument-hooks to protocol version 3. Integrations timestamp their
OpenTelemetry spans and logs with the monotonic clock of the benchmark
markers, shifted by `instrument_hooks_monotonic_offset` to read as wall
clock times. In analysis mode, the hooks now send that offset to the
runner in a new `SetTelemetryMonotonicOffset { pid, offset_ns }` command,
so that the runner can place the telemetry back on the clock of the
markers.

The runner accepts protocol version 3 and acknowledges the command in
every executor. Executors that don't collect telemetry ignore it.

Refs COD-3863
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the files holding the OpenTelemetry spans and logs the benchmarks
emit, shared by the runner that writes them and the parsers that read
them: `otel/spans.msgpack.zst` and `otel/logs.msgpack.zst` in the
profile folder.

Each file is a zstd stream of positional MessagePack records, one per
span or log, like the memtrack artifacts. Rows are flattened for the
tables they are loaded into: ids as hex, attribute values as text, the
benchmark in its own field, and the span or log time relative to the
benchmark start when known. zstd runs with a 16 MB window and
long-distance matching, since rows repeat their resource attributes far
apart.

Versions follow `UnwindData`: each record is wrapped in a versioned enum
(`SpanRecord`, `LogRecord`), `SpanRow` and `LogRow` alias the current
version, and older versions are upgraded through `From`. Version 1
fixtures must keep decoding.

`TelemetryReader` decodes a file from any `Read`, such as an archive
entry, as its bytes arrive. Unlike a truncated memtrack stream, a file
cut in the middle of a row ends with an error.

Refs COD-3863
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `codspeed run --mode telemetry`, an experimental mode collecting the
OpenTelemetry spans and logs the benchmarks export. It is a separate
pass, so nothing has to keep the export out of the measured runs, and
the benchmarks run once each instead of once per walltime iteration.

The mode runs on a new analysis executor: it runs the benchmarks once
each without a profiler, answering the integration as an analysis mode,
with analysis instruments hooked onto it. An instrument sets up before
the process is spawned, gets the integration's start and stop profiler
commands, and stops once the process exits, with the benchmark markers.
The executor saves those markers as `ExecutionTimestamps.msgpack` in the
profile folder, and fails when no benchmark ran, unless `--allow-empty`
is set. Uploads carry `executor: "analysis"` and the enabled
instruments, so later analysis instruments only add an instrument.

The telemetry mode enables the telemetry instrument:

- It starts an OTLP/HTTP receiver on a free localhost port, passed to
  the benchmark in CODSPEED_OTEL_ENDPOINT. The receiver accepts
  protobuf and JSON exports, plain or gzipped, and writes each trace
  and log request to disk as received. Metrics are acknowledged and
  dropped.
- Once the process exits, the requests are decoded and written as the
  runner-shared telemetry files, one row per span or log, formatted
  like the OpenTelemetry Collector's ClickHouse exporter. A request that
  fails to decode is skipped with a warning.
- Each row is placed on the clock of its benchmark: the clock offset the
  benchmark process sent in `SetTelemetryMonotonicOffset` is subtracted
  from its timestamp, and the result is measured from the start marker
  of the benchmark named by its `codspeed.benchmark.uri` attribute. The
  offset is left out when either is missing. With several benchmark
  processes, the first offset is used, with a warning when another
  differs from it by more than 1 ms.

`CODSPEED_RUNNER_MODE` is `telemetry` in this mode, which
`cargo codspeed run` reads to pick its telemetry build. exec-harness
runs it like memory mode.

Refs COD-3863
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Build the divan `otel_example` of a codspeed-rust checkout next to this
repository with `cargo codspeed build -m telemetry`, run it through
`codspeed run --mode telemetry`, and read back its telemetry files: they
hold the benchmark's spans and logs, each tagged with a benchmark that
ran and placed within its benchmark's sample.

The test is ignored by default since CI has no codspeed-rust checkout.
Run it with `cargo test --test otel -- --ignored`.
`scripts/run-otel-example.sh` runs the same example and uploads it.

Refs COD-3863
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lvaroqui
lvaroqui force-pushed the cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet branch from 33db967 to cf0d278 Compare October 20, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant